Shelf label detection device, shelf label detection method, and shelf label detection program

ABSTRACT

The present disclosure accurately detects the position of a shelf label disposed on a display shelf. A shelf label detection device may be provided with a shelf label position correction unit and a shelf label position identification unit. The shelf label position correction unit corrects a manual shelf label position set including a shelf label position that has been set in advance for a reference camera image, using an automatic shelf label position set which includes a shelf label position detected from a monitoring camera image using image recognition, thereby generating a corrected shelf label position set. The shelf label position identification unit uses the corrected shelf label position set to identify the shelf label position in the monitoring camera image.

BACKGROUND Technical Field

The present disclosure relates to a shelf label detection device, ashelf label detection method, and a shelf label detection program.

Description of the Related Art

In the retail industry, business has become more efficient by, forexample, systematizing the management of a stock and/or sales ofproducts. For the stock management, for example, a system that monitorsa stock of products based on video of a product display area (e.g., adisplay shelf) obtained by a monitoring camera may be used.

LITERATURE LIST Patent Literature

-   PTL 1-   Japanese Patent Application Laid-Open No. 2015-103153-   PTL 2-   Japanese Patent No. 6112436

SUMMARY

However, for example, when a shelf row (i.e., an area or region to bemonitored) to be included in a monitoring target is not appropriatelyset for video including a display shelf, it may be difficult toaccurately monitor the stock quantity of products (e.g., shortage orlacking of products) in some cases.

Thus, after all, a person may be required to visually inspect a stock ofindividual products. With visual inspection, in some cases, there may bea discrepancy between the number of individual products (physical stockquantity) actually displayed in a display area and, for example, thenumber of stock quantity of individual products (theoretical stockquantity) managed by a stock management system.

When there is a discrepancy between the physical stock quantity and thetheoretical stock quantity, for example, a person may be required tomanually correct stock management data, such as the theoretical stockquantity in accordance with the physical stock quantity confirmed byvisual inspection, and the correction takes time.

Since it is burdensome for a stock manager to make such correction everyday and/or every hour, for example, a specific product may remainlacking. When the product remains lacking, a product selling opportunityis lost, and as a result, customers may have a negative image in somecases.

Thus, a stock monitoring system that automatically monitors the stockquantity of products has been desired. However, in order to implementthis system, it is necessary to accurately detect the position of ashelf label included in camera video.

One non-limiting and exemplary embodiment facilitates providing a shelflabel detection device, a shelf label detection method, and a shelflabel detection program available to detect a shelf label positionaccurately from camera video of a display shelf.

Solution

In one general aspect, the techniques disclosed here related to a shelflabel detection device includes: an obtainer that obtains first cameravideo and second camera video including an image of a shelf labelarranged in a display shelf; a corrector that corrects a first shelflabel position set to generate a corrected shelf label position set byusing a second shelf label position set, the first shelf label positionset including a shelf label position that is set by a user in advance inthe first camera video, the second shelf label position set including ashelf label position for the image of the shelf label detected from thesecond camera video through video recognition; and an determiner thatdetermines the shelf label position in the second camera video by usingthe corrected shelf label position set.

In one general aspect, the techniques disclosed here related to a shelflabel detection method according to one aspect of the present disclosureincludes: obtaining first camera video and second camera video includingan image of a shelf label arranged in a display shelf; correcting afirst shelf label position set to generate a corrected shelf labelposition set by using a second shelf label position set, the first shelflabel position set including a shelf label position that is set by auser in advance in the first camera video, the second shelf labelposition set including a shelf label position for the image of the shelflabel detected from the second camera video through video recognition;and determining the shelf label position in the second camera video byusing the corrected shelf label position set.

In one general aspect, the techniques disclosed here related to a shelflabel detection program according to one aspect of the presentdisclosure causes a computer to execute a process including: obtainingfirst camera video and second camera video including an image of a shelflabel arranged in a display shelf; correcting a first shelf labelposition set to generate a corrected shelf label position set by using asecond shelf label position set, the first shelf label position setincluding a shelf label position that is set by a user in advance in thefirst camera video, the second shelf label position set including ashelf label position for the image of the shelf label detected from thesecond camera video through video recognition; and determining the shelflabel position in the second camera video by using the corrected shelflabel position set.

Additional benefits and advantages of the disclosed embodiments willbecome apparent from the specification and drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the specification and drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

Advantageous Effects

In one general aspect, it is possible to accurately detect the positionof a shelf label included in camera video.

Further advantages and benefits in an aspect of the present disclosurewill become apparent from the specification and the drawings. Althoughthe advantages and/or benefits are provided by some embodiments andfeatures illustrated in the specification and the drawings, not all ofthe advantages and the benefits need to be provided to obtain one ormore such features.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a configuration example of a stock management systemaccording to an embodiment;

FIG. 2 is a schematic front view of a display shelf to which shelflabels are attached according to the embodiment;

FIG. 3 is an enlarged view of part of the front view of the displayshelf illustrated in FIG. 2;

FIG. 4 illustrates an example of shelf allocation information accordingto the embodiment;

FIG. 5 is a block diagram illustrating a configuration example of acomputer illustrated in FIG. 1;

FIG. 6 is a flowchart illustrating an operation example of the computer(stock monitoring device) illustrated in FIGS. 1 and 5;

FIG. 7 illustrates an example in which shelf labels are detected throughvideo recognition in the stock monitoring device illustrated in FIGS. 1and 5;

FIG. 8 illustrates an example in which a monitoring area is set by thestock monitoring device illustrated in FIGS. 1 and 5;

FIG. 9 illustrates an example of a flow of a shelf label associatingprocess performed by the stock monitoring device illustrated in FIGS. 1and 5;

FIG. 10 illustrates an example of a flow of a shelf label associatingprocess performed by the stock monitoring device illustrated in FIGS. 1and 5;

FIG. 11 illustrates an example of a flow of a shelf label associatingprocess performed by the stock monitoring device illustrated in FIGS. 1and 5;

FIG. 12 illustrates an example of a flow of a shelf label associatingprocess performed by the stock monitoring device illustrated in FIGS. 1and 5;

FIG. 13 illustrates an example of a flow of a shelf label associatingprocess performed by the stock monitoring device illustrated in FIGS. 1and 5;

FIG. 14 illustrates an example in which a monitoring area is set by thestock monitoring device illustrated in FIGS. 1 and 5;

FIGS. 15A and 15B illustrate examples of a search order in the shelflabel detection process performed by the stock monitoring deviceillustrated in FIGS. 1 and 5;

FIG. 16 illustrates an example in which a monitoring area is set by thestock monitoring device illustrated in FIGS. 1 and 5 by using the shelfallocation information;

FIG. 17 illustrates an example in which a shelf label detection resultobtained by the stock monitoring device illustrated in FIGS. 1 and 5 iscorrected;

FIG. 18 illustrates the example in which the shelf label detectionresult obtained by the stock monitoring device illustrated in FIGS. 1and 5 is corrected;

FIG. 19 illustrates the example in which the shelf label detectionresult obtained by the stock monitoring device illustrated in FIGS. 1and 5 is corrected;

FIG. 20 illustrates an example of applying a PTZ (pan, tilt, and zoom)camera as a camera illustrated in FIG. 1;

FIG. 21 illustrates an example in which recognition models used forpattern matching of video recognition differ for each imaging directionin FIG. 20;

FIGS. 22A to 22C illustrate examples in which stepwise stock levels areset by the stock monitoring device illustrated in FIGS. 1 and 5;

FIG. 23 illustrates an example in which stock levels are detected in thedepth direction of shelf rows in the stock monitoring device illustratedin FIGS. 1 and 5;

FIG. 24 illustrates an example in which stock levels are detected in thedepth direction of shelf rows in the stock monitoring device illustratedin FIGS. 1 and 5;

FIGS. 25A and 25B illustrate an example of a mobile object removingprocess performed by the stock monitoring device illustrated in FIGS. 1and 5;

FIGS. 26A to 26D illustrate issues in a case where a shelf labelposition is determined only by a known video recognition techniqueregarding Embodiment 2;

FIGS. 27A and 27B illustrate issues in a case where shelf labelpositions are manually input regarding Embodiment 2;

FIG. 28 illustrates an outline of a method for determining shelf labelpositions according to Embodiment 2;

FIG. 29 is a block diagram illustrating a configuration example of ashelf label detection device according to Embodiment 2;

FIG. 30 is a flowchart illustrating an operation example of the shelflabel detection device according to Embodiment 2;

FIG. 31 is a flowchart illustrating an operation example of the shelflabel detection device according to Embodiment 2;

FIG. 32 is a flowchart illustrating an example of a process performed bya shelf label position corrector;

FIG. 33 illustrates an example in which movement amounts (offsetamounts) are calculated;

FIG. 34 is a block diagram illustrating a configuration example of ashelf label detection device according to Embodiment 3;

FIG. 35 illustrates a specific example of a process in which a firstupdate request alert is issued;

FIG. 36 is a flowchart illustrating an example of the process in whichthe first update request alert is issued;

FIG. 37 illustrates a specific example of a process in which a secondupdate request alert is issued;

FIG. 38 is a flowchart illustrating an example of the process in whichthe second update request alert is issued;

FIG. 39 illustrates a specific example of a process for automaticallymodifying a manual shelf label position set; and

FIG. 40 is a flowchart illustrating an example of the process forautomatically modifying the manual shelf label position set.

DETAILED DESCRIPTION

Now, embodiments will be described below in detail with reference to thedrawings as appropriate. Detailed description more than necessary may beomitted. For example, detailed description of a well-known matter andrepeated description of substantially the same configuration may beomitted. This will prevent unnecessarily redundant description below andwill help easy understanding of a person skilled in the art.

The attached drawings and the following description are provided for aperson skilled in the art to fully understand the present disclosure andare not intended to limit the subject described in Claims.

Embodiment 1

FIG. 1 illustrates a configuration example of a stock management systemaccording to an embodiment. Stock management system 1 illustrated inFIG. 1 may include, for example, camera 10 and computer 20. The “stockmanagement system” may be, in other words, a “stock monitoring system”.

Camera 10 is, for example, arranged in a store selling products andcaptures video including an area where products are displayed, forexample, an area where display shelf 50 for products is arranged. Thetype of business or business condition of the store where display shelf50 is arranged and types of products handled by the store are notlimited.

For example, display shelf 50 may be provided in a supermarket, aconvenience store, a department store, a mass merchandiser, a discountstore, or a shop or a selling booth (or a selling corner) arranged inany facility. In addition, display shelf 50 may also be provided outsidenot only inside.

Camera 10 may be a dedicated camera that images an area includingdisplay shelf 50 or may be a camera that is also used for anotherpurpose or usage, such as a surveillance camera. Also, multiple cameras10 may be provided in stock management system 1.

An imaging target (i.e., “monitoring target”) by camera 10 and displayshelf 50 may correspond to each other in a one-to-one relationship, aone-to-many relationship, or a many-to-one relationship. For example,single display shelf 50 may be an imaging target of single camera 10, ormultiple display shelves 50 may be an imaging target of single camera10.

For example, by using camera 10 whose imaging direction and/or angle ofview can be changed, such as a PTZ camera, multiple different displayshelves 50 may be included in an imaging target of single PTZ camera 10.Alternatively, different regions or spaces of single display shelf 50may be included in an imaging target of single PTZ camera 10.

For example, when it is difficult to include the width or height ofsingle display shelf 50 in camera video of single camera 10, bycontrolling the imaging direction of one or more PTZ cameras 10 to bevariable, a plurality of regions or spaces with different widths orheights may be captured in a camera image.

In the above manner, by using a PTZ camera as camera 10, it isunnecessary to arrange camera 10 for each display shelf 50 or eachdifferent region or space of display shelf 50. Thus, it is possible toreduce the number of cameras 10 arranged in stock management system 1.

Computer 20 illustrated in FIG. 1 is an example of an informationprocessing device and may be a personal computer (PC) or a server. Theserver may include a cloud server. Computer 20 and camera 10 are, forexample, connected to each other with a wire or wirelessly andcommunicate with each other. In addition, computer 20 and camera 10 maybe connected to each other via a network.

The term “network” may be a wired network or a wireless network.Examples of the wired network include an intranet, the Internet, and awired local area network (LAN). Examples of the wireless network includewireless LAN.

Computer 20, for example, receives video data obtained by camera 10(hereinafter also abbreviated to “camera video”) and analyzes thereceived camera video. For example, through video recognition of cameravideo of display shelf 50, computer 20 monitors a stock of products indisplay shelf 50 and detects shortage or lacking of a product.

“Video recognition” may also be referred to as “image recognition”. Inaddition, detecting the shortage or lacking of a product maycollectively be referred to as “detecting of lacking” for convenience.“Detecting” may also be referred to as “sensing”. In addition, a devicefrom which camera video is transmitted may be camera 10 or, for example,a recording device that records video data obtained by camera 10.

Monitoring a stock of products may include detecting of the position ofa shelf label attached to display shelf 50. The shelf label mayindicate, for example, information on a product (hereinafter referred toas “product information”), such as a product name and/or price. Theshelf label may be a paper shelf label or an electronic shelf label. Theelectronic shelf label may be formed as a liquid crystal display or thelike or may be formed as electronic paper or the like. In addition, theelectronic shelf label may have a wireless communication function or thelike, and presented information may be rewritable by remote control.“Shelf label” may also be referred to as another name, such as a shelftag, a shelf card, or a bin tag.

For example, through video recognition of camera video, computer 20detects the position of a shelf label attached to display shelf 50, and,based on the detected position of the shelf label, may set a region orspace for monitoring a stock in display shelf 50.

Hereinafter, the region or space for monitoring a stock of products maycollectively be referred to as “monitoring area” or “monitoring region”.An example of setting the monitoring area in display shelf 50 based onthe position of shelf label 51 detected through video recognition willbe described later.

FIG. 2 is a schematic front view of display shelf 50 to which shelflabels 51 are attached. FIG. 3 is an enlarged view of part of the frontview of display shelf 50 illustrated in FIG. 2. As a non-limitingexample, FIGS. 2 and 3 illustrate a form in which a display space ofdisplay shelf 50 is divided into four spaces in the height direction ofdisplay shelf 50 by three shelf boards 52. In FIGS. 2 and 3, a regionsurrounded by a dotted frame represents a condition in which product 70is lacking.

In addition, FIG. 3 also illustrates an example in which, computer 20performs video recognition on the video data including display shelf 50obtained by camera 10 and detects the shortage or lacking of a productin the monitoring area set based on the position of detected shelf label51. The detected information is output by computer 20.

The display space divided in the height direction of display shelf 50may also be referred to as “shelf row”. FIG. 2 illustrates an examplefocusing on two shelf rows. The manner of dividing the display space indisplay shelf 50 is not limited. In addition to the height direction ofdisplay shelf 50, the display space may also be divided in the widthdirection of display shelf 50.

Shelf label 51 may be attached to any position of the display shelf 50at which a correspondence relationship with product 70 to be displayedis visually recognizable. For example, shelf label 51 may be attached toshelf board 52. In each display space, product 70 is, for example,displayed in a region or space corresponding to the position ofcorresponding shelf label 51 (space above shelf label 51 in the examplein FIGS. 2 and 3).

For setting the monitoring area and/or for monitoring a stock, inaddition to position information of detected shelf label 51, informationon shelf allocation (hereinafter referred to as “shelf allocationinformation”) may be used. “Shelf allocation” indicates, for example, aplan about products and numbers thereof to be displayed in (or allocatedto) each display space of display shelf 50.

FIG. 4 illustrates an example of shelf allocation information 400. Asillustrated in FIG. 4, shelf allocation information 400 may include, forexample, information indicating the position at which product 70 is tobe displayed and information on product 70 to be displayed at theposition. The information indicating the position at which product 70 isto be displayed may be referred to as “display position information”,and the information on product 70 may be referred to as “productinformation” in the following description.

As a non-limiting example, the display position information may includeinformation indicating any one or more of a store number, a floornumber, an aisle number, a shelf number, a shelf row number, and adisplay position in a shelf row.

The product information may include, for example, information by whichindividual product 70 can be determined or identified, such as the typeor content of product 70. Non-limiting examples of the information bywhich product 70 can be determined or identified include a product name,such as “XXX pasta” or “YYY curry”, and a product code.

The product information may include, for example, information indicatingthe size (at least one of the width, height, and depth) of product 70,and information indicating the number of products 70 to be displayed, inother words, information indicating a display number. In addition, theproduct information may further include information indicating the priceof product 70.

For example, “display number” may indicate the number of products 70 tobe displayed in one or more of the width direction, the heightdirection, and the depth direction of a shelf row. Based on either theinformation indicating the size of product 70 or the informationindicating the display number of product 70, or both, for example, it ispossible to determine a space or region occupied by multiple products 70in a shelf row with higher accuracy.

Accordingly, with referring to shelf allocation information 400,computer 20 is possible to increase the accuracy for setting themonitoring area based on the shelf label position, and as a result, theaccuracy for detecting lacking of product 70 can be increased.

In addition, based on shelf allocation information 400, computer 20 maycorrect the detecting result (e.g., the shelf label position) of shelflabel 51 attached to display shelf 50 based on video recognition.Correction of the shelf label position may include, for example,correction of failure in detecting shelf label 51 through videorecognition based on shelf allocation information 400.

Examples of setting the monitoring area and examples of correcting theshelf label position by using shelf allocation information 400 will bedescribed later.

In a case where shelf allocation information 400 is not used, specificinformation on the short or lacking of product 70 in the monitoring areamay be unavailable. However, from a result of image recognition of themonitoring area set based on the shelf label position (e.g., a ratio ofa background image appearing in the monitoring area), it is possible todetect the position and the shelf row in which product 70 in lacking isshort or lacking. By additionally using shelf allocation information 400in such a detecting process, it is possible to determine specificinformation on what product 70 is in lacking. For example, it ispossible to provide information such as “XXX pasta is lacking” or “XXXpasta in a x-th row and a y-th column is lacking”.

(Configuration Example of Computer 20)

Next, a configuration example of computer 20 will be described withreference to FIG. 5. As illustrated in FIG. 5, for example, computer 20may include processor 201, input device 202, output device 203, memory204, storage 205, and communicator 206.

Processor 201 controls the operation of computer 20. Processor 201 is anexample of a circuit or device having an arithmetic capability. Asprocessor 201, for example, at least one of a central processing unit(CPU), a micro processing unit (MPU), and a graphics processing unit(GPU) may be used.

Input device 202 may include, for example, at least one of a keyboard, amouse, an operation button, and a microphone. Through input device 202,data or information may be input to processor 201.

Output device 203 may include, for example, at least one of a display(or a monitor), a printer, and a speaker. For example, the display maybe a touch-panel display. The touch-panel display may correspond to bothinput device 202 and output device 203.

Memory 204 stores, for example, a program executed by processor 201 anddata or information processed in accordance with the execution of theprogram. Memory 204 may include a random access memory (RAM) and a readonly memory (ROM). The RAM may be used as a work memory of processor201. The term “program” may also be referred to as “software” or“application”.

Storage 205 stores a program executed by processor 201 and data orinformation processed in accordance with the execution of the program.Storage 205 may store shelf allocation information 400 described above.Shelf allocation information 400 may be stored in storage 205 in advanceor may be, for example, provided from a shelf allocation system (notillustrated) that manages shelf allocation information 400 and stored instorage 205.

Storage 205 may include a semiconductor drive device such as a hard diskdrive (HDD) or a solid state drive (SSD). In addition to or in place ofa semiconductor drive device, a non-volatile memory, such as a flashmemory, may be included in storage 205.

The program may include a stock monitoring program that monitors a stockof product 70 through video recognition, as described above. All or partof a program code composing the stock monitoring program may be storedin memory 204 and/or storage 205 or may be incorporated in part of anoperating system (OS).

The program and/or data may be provided in the form of being recorded ona recording medium that can be read by computer 20. Examples of therecording medium include a flexible disk, a CD-ROM, a CD-R, a CD-RW, anMO, a DVD, a Blu-ray disc, a portable hard disk, and the like. Inaddition, a semiconductor memory such as a universal serial bus (USB)memory is also an example of the recording medium.

In addition, the program and/or data may be, for example, provided(downloaded) to computer 20 from a server (not illustrated) via acommunication line. For example, the program and/or data may be providedto computer 20 through communicator 206 and stored in memory 204 and/orstorage 205. In addition, the program and/or data may be provided tocomputer 20 through input device 202 and may be stored in memory 204and/or storage 205.

Communicator 206 includes, for example, communication interface (IF) 261for communication with camera 10. Communication IF 261 may be any of awired interface and a wireless interface.

For example, communication IF 261 receives video data obtained by camera10. The received video data is, for example, stored in memory 204 and/orstorage 205 through processor 201. When camera 10 is a PTZ camera, forexample, communicator 206 may communicate with PTZ camera 10 inaccordance with an instruction from processor 201 to control the imagingdirection and/or angle of view of PTZ camera 10.

In addition, communicator 206 may further include communication IF 262for communication with “another computer” (not illustrated) differentfrom computer 20. The “other computer” may be, for example, a serverconnected to a wired or wireless network or a user terminal connected toa wired or wireless network. The “other computer” may correspond to acomputer in the above-described shelf allocation system.

The user terminal may be owned by, for example, a stock manager ofproduct 70. Non-limiting examples of the user terminal include a PC, amobile phone (including a smartphone), and a tablet terminal. The userterminal may be provided with information related to stock management orstock monitoring of products.

Processor 201 may, for example, read and execute the stock monitoringprogram stored in memory 204 and/or storage 205 to cause computer 20 tofunction as a stock monitoring device that monitors a stock of product70 through video recognition.

For example, processor 201 executes the stock monitoring program, andthereby, stock monitoring device 20 illustrated in FIG. 5 is embodied.Stock monitoring device 20 includes shelf label detector 211, monitoringarea setter 212, lacking detector 213, and output section 214.Optionally, in stock monitoring device 20, either shelf labelassociating section 215 or detected shelf label corrector 216, or both,may be embodied in accordance with the execution of the stock monitoringprogram.

Shelf label detector 211, for example, detects shelf label 51 includedin camera video through video recognition of camera video including theentire or part of display shelf 50. For example, by using a templateimage corresponding to the shape and/or color of shelf label 51, shelflabel detector 211 may perform pattern matching of the camera video todetect shelf label 51.

Monitoring area setter 212, for example, sets the monitoring area indisplay shelf 50 based on the position of shelf label 51 detected byshelf label detector 211.

Lacking detector 213 is an example of a monitoring section and, forexample, monitors a stock (or may be referred to as “display condition”)of product 70 in display shelf 50 based on a change in videocorresponding to the presence or absence of a product in the monitoringarea set by monitoring area setter 212.

For example, by using, as the template image, background image thatappears in the monitoring area when product 70 is short or lacking,lacking detector 213 may perform pattern matching of the camera video ofthe monitoring area to detect shortage or lacking of product 70.

The shape of shelf label 51 may differ in the camera video depending onthe position where camera 10 is arranged and/or the imaging directionthereof. For example, the shape of shelf label 51 differs between cameravideo capturing display shelf 50 from the front and camera videocapturing display shelf 50 obliquely from the front.

Further, the background image that appears in the camera video whenproduct 70 is short or lacking may differ depending on the positionwhere camera 10 is arranged and/or the imaging direction thereof(hereinafter also collectively referred to as “camera position” forconvenience).

For example, in camera video capturing display shelf 50 from the front,an image of a backboard seen from the front, the back board beinglocated in the back surface of display shelf 50, may correspond to thebackground image. In addition, in camera video capturing display shelf50 obliquely from above, for example, a surface of shelf board 52 onwhich product 70 is displayed may correspond to the background image. Incamera video capturing display shelf 50 obliquely from a side, forexample, a surface of a partition (not illustrated) that partitions theshelf row in the horizontal direction may correspond to the backgroundimage.

In the above manner, the shape of shelf label 51 and/or the backgroundimage differs depending on the camera position. Thus, a template image(i.e., “recognition model”) used for pattern matching of videorecognition corresponding to the camera position may be prepared. Inthis embodiment, the recognition model is a template image, and theshape of shelf label and/or the background image is recognized bypattern matching. However, other implementation methods are possible.For example, a trained model generated by machine learning of each ofthe shelf label and/or the background image may be used as therecognition model to recognize the shelf label and/or the backgroundimage.

For example, when camera 10 is arranged at a plurality of positions,and/or when the imaging direction is variable such as in PTZ camera 10,a plurality of template images may be prepared. The template images are,for example, stored in storage 205 and read by processor 201 at anappropriate time.

Output section 214 is, an example of a notification informationgenerator that generates and outputs information to be presented (e.g.,sent as a notification) to, for example, the stock manager. Outputsection 214, for example, generates notification information including adetection result of lacking detector 213 and/or information based on thedetection result and outputs the notification information to outputdevice 203 and/or communicator 206.

As a non-limiting example, information as a notification of detecting oflacking of product 70 (also referred to as “notification information”,“lack information”, or “alert information”) may be output to a displayand/or a printer, which is an example of output device 203.

Output section 214 may generate the notification information based oninformation in which either one or both of a detection result of lackingdetector 213 and a detection result of shelf label detector 211 is/areassociated with shelf allocation information 400.

For example, based on information in which the detection result oflacking detector 213 is associated with shelf allocation information400, output section 214 can generate the notification informationincluding the position related to detecting of lacking of product 70(lack area) and/or a product name of product 70 for which lacking isdetected.

The notification information may be sent to, for example, the “othercomputer” through communicator 206. Email may be used to send thenotification information through communicator 206.

Shelf label associating section 215, for example, associates each shelflabel 51 detected by shelf label detector 211 with shelf allocationinformation 400 (e.g., product information).

When shelf allocation information 400 includes, as an example of theproduct information, information on the size and the display number ofproduct 70, for example, based on the product information, detectedshelf label corrector 216 may correct the shelf label position detectedby shelf label detector 211.

The configuration of computer (stock monitoring device) 20 illustratedin FIG. 5 is an example. Hardware components and/or functional blocks instock monitoring device 20 may be increased or decreased as appropriate.For example, hardware components and/or functional blocks may be added,deleted, divided, or integrated in stock monitoring device 20 asappropriate.

(Operation Example)

Next, an operation example of stock monitoring device 20 will bedescribed.

As illustrated in FIG. 6, stock monitoring device 20, for example,obtains camera video (S11) and analyzes the obtained camera video inshelf label detector 211, for example, thereby detecting shelf label 51included in the camera video (S12). For example, as illustrated in FIG.7 with thick frame 500, eight shelf labels 51 are detected through videorecognition.

In the example in FIG. 7, two shelf labels 51 are detected on shelfboard 52 in a m-th row from the bottom, and on each of shelf boards 52(in (m+1)-th and (m+2)-th rows from the bottom) located thereabove,three shelf labels 51 are detected. The m is an integer greater than orequal to 1.

Positions of two shelf labels 51 detected on shelf board 52 in the m-throw from the bottom are, for example, n-th and (n+1)-th positions (n isan integer greater than or equal to 1) from the left. Similarly,positions of three shelf labels 51 detected on shelf board 52 in the(m+1)-th row from the bottom are, for example, n-th, (n+1)-th, and(n+2)-th positions from the left. Positions of three shelf labels 51detected on shelf board 52 in the (m+2)-th row from the bottom are, forexample, n-th, (n+1)-th, and (n+2)-th positions from the left. The “m-throw and n-th” position may be denoted as a “m-th row, n-th position” or“m-th row, n-th column” position.

When multiple cameras 10 are included in stock management system 1 orwhen a PTZ camera is used as camera 10, stock monitoring device 20 may,for example, obtain information indicating the camera position and mayprovide the information to shelf label detector 211 (S11 a in FIG. 6).

The information indicating the camera position may be, for example,associated with shelf allocation information 400 in advance. Forexample, based on the information indicating the camera position andshelf allocation information 400, shelf label detector 211 may identifycamera 10 and its imaging direction of the camera video and may set arecognition model appropriate for detection of shelf label 51 throughvideo recognition. Examples of association between the informationindicating the camera position and shelf allocation information 400 willbe described later.

In response to detection of shelf label 51 (S12 in FIG. 6), stockmonitoring device 20 sets, by using monitoring area setter 212, forexample, a monitoring area for monitoring a stock of product 70 (S13 inFIG. 6). For example, monitoring area setter 212 sets one of detectedshelf labels 51 as reference shelf label 51. As a non-limiting example,FIG. 7 illustrates a form in which lower left (first row, firstposition) shelf label 51 is set as reference shelf label 51.

Reference shelf label 51 may be autonomously set by stock monitoringdevice 20 (e.g., monitoring area setter 212) or may be designated(manually designated) by a user (e.g., a stock manager) of stockmonitoring device 20.

For example, reference shelf label 51 can be autonomously set throughvideo recognition by differentiating the external appearance ofreference shelf label 51 from the others, such as color (e.g., framecolor) and/or shape, from the external appearance of the other shelflabels 51.

Alternatively, information of “x-th row, y-th position” shelf label 51may be input to monitoring area setter 212 through input device 202 asinformation specifying reference shelf label 51. In addition, manualspecification may be used for compensating for autonomous setting.

In response to setting of reference shelf label 51, monitoring areasetter 212 may, for example, detect shelf label 51 adjacent to referenceshelf label 51 in the vertical direction and/or horizontal directionbased on shelf allocation information 400 (e.g., the display positioninformation) illustrated in FIG. 4.

Shelf label 51 that is adjacent in the vertical direction and/orhorizontal direction may be referred to as “adjacent shelf label 51” forconvenience. “Detection” of adjacent shelf label 51 may also be referredto as “search” or “retrieval” for adjacent shelf label 51.

Based on the distance between detected shelf labels 51, monitoring areasetter 212 may set the monitoring area. An example for setting themonitoring area is illustrated in FIG. 8.

In FIG. 8, for example, when m-th (=1) row, n-th (=1) position shelflabel 51 is set as a reference shelf label, monitoring area setter 212obtains the distance between reference shelf label 51 and adjacent shelflabel 51.

For example, distance Rx between m-th row, n-th position reference shelflabel 51 and m-th row, (n+1)-th position shelf label 51 on the right isdetected, and distance Ry between reference shelf label 51 and (m+1)-throw, n-th position shelf label 51 thereabove is detected.

Monitoring area setter 212, for example, sets monitoring area MA (seethe dotted frame) having a size and a shape determined by distances Rxand Ry as a monitoring area for first-row, first position shelf label51. Regarding other shelf labels 51, monitoring area setter 212 detectsdistances Rx and Ry between adjacent shelf labels 51 so as to setmonitoring areas MA for respective shelf labels 51 detected throughvideo recognition.

In the above manner, for each shelf label 51 detected through videorecognition, monitoring area setter 212 calculates a distance betweenshelf label 51 and an adjacent shelf label, and sets monitoring area MAcorresponding to each shelf label 51 based on the calculated distance.Thus, it is possible to accurately set monitoring area MA for product 70to be displayed corresponding to shelf label 51.

The shape of monitoring area MA may be rectangular or circular includingelliptic. In addition, shelf label 51 may be arranged in accordance witha predetermined rule (or criteria) for each space where product 70 is tobe displayed. For example, shelf label 51 may be arranged in accordancewith such a certain rule that shelf label 51 is arranged at the lowerleft of product 70. When the rule is defined, based on the position ofeach shelf label 51 or the distance between shelf labels 51 and therule, it is possible to set monitoring area MA accurately. For example,when shelf label 51 is arranged at the lower left of product 70, themonitoring area would be present on the right of shelf label 51.

In addition, when there is a shelf row in which no shelf label 51 isarranged (e.g., the top shelf row), by arranging dummy shelf label 51,the distance between shelf labels 51 including dummy shelf label 51 maybe determined. When obtaining distance Ry above the top shelf row byusing dummy shelf label 51, for example, dummy shelf label 51 may bearranged on “POP” arranged for the shelf. “POP” is an advertisingmedium, such as paper on which a product name, product price,advertising slogan, explanation, and/or illustration is written.Alternatively, for distance Ry for a shelf row in which shelf label 51is not arranged, the distance between shelf labels 51 may be obtainedbased on the distance from another shelf row (e.g., a lower shelf row).For example, for shelf label 51 with no adjacent shelf label 51,distance Ry calculated for another shelf label 51 may be reused. Forexample, Ry for an immediately below shelf row with little distortedperspective may be reused, or Ry may be estimated by converting Ry foranother shelf row by taking camera parameters into account.Alternatively, for a shelf row with no shelf label 51, the distancebetween shelf labels 51 may be manually set.

In addition, for shelf label 51 located at the right end of a shelf row,when another adjacent display shelf 50 is present, the distance fromshelf label 51 attached to the display shelf 50 may be determined as thedistance between shelf labels 51. Alternatively, for shelf label 51located at the right end of a shelf row, the distance from an image endof camera video may be set as the distance between shelf labels 51.

Shelf allocation information 400 may include, for example, informationon one or more of the width, height, depth, and display number ofproduct 70, which is a monitoring target. In this case, based on thisinformation, detected shelf label corrector 216 may correct the shelflabel position detected by shelf label detector 211 (S12 a in FIG. 6).

In other words, shelf allocation information 400 may be used forchecking the accuracy and/or correcting a detection result of shelflabel 51 with video recognition. With correction of the position ofshelf label 51, it is possible to increase the accuracy for detecting oflacking of product 70, which is a monitoring target. Examples forcorrecting the shelf label position will be described later withreference to FIGS. 17 to 19.

After the setting of monitoring area MA, stock monitoring device 20, byusing lacking detector 213, for example, detects a region where product70 is short or lacking in monitoring area MA by pattern matching betweencamera video and background image of individual monitoring area MA (S14in FIG. 6).

For example, in FIG. 8, two products 70 are displayed in a space wherethree products 70 can be displayed for m-th row, n-th column shelf label51. Thus, background image corresponding to single product 70 appears incamera video of monitoring area MA.

When the background image appearing in monitoring area MA correspondswith the template image by pattern matching, lacking detector 213detects that product 70 is not displayed in a region where thebackground image appears. In other words, shortage or lacking of product70 corresponding to shelf label 51 is detected.

In response to detection of shortage or lacking of product 70, stockmonitoring device 20, by using output section 214 for example, generatesand outputs information as a notification of detecting of lacking ofproduct 70 (S15 in FIG. 6). The information as a notification ofdetecting of lacking of product 70 may include text information and/orsound information or may include, for example, information fordisplaying a region regarding detecting of lacking in an emphasizedmanner on a display displaying the camera video.

Non-limiting examples of display in an emphasized manner include thefollowing. The following display in an emphasized manner may be combinedas appropriate.

In camera video, the color of the region regarding detecting of lacking(hereinafter also collectively referred to as “lack area” forconvenience) is changed to more outstanding color (emphasized color)than the color of the other regions.

The lack area is blinked.

The lack area is displayed in a solid frame or a dotted frame. Color(emphasized color) may be used for the solid frame or dotted frame.

The solid frame or dotted frame for the lack area is blinked.

As described above, according to stock monitoring device 20, based onthe position of shelf label 51 detected through video recognition ofcamera video including display shelf 50, the monitoring area for displayshelf 50 in the camera video is set. Thus, it is possible toappropriately set the monitoring area to the camera video includingdisplay shelf 50 without a manual operation. When a reference shelflabel is manually designated, it may be unavailable to completely removea manual operation for setting the monitoring area; however, the settingof monitoring area can be almost automated, and also, an accuratereference shelf label can be set. This enables setting of the monitoringarea more accurately and more quickly than in the related art.

In other words, a display space of a monitoring target in display shelf50 can be automatically set. Accordingly, for example, even when arelative position relationship between display shelf 50 and camera 10varies due to an external cause such as vibrations, detection of shelflabel 51 with video recognition enables reconfiguring of an appropriatemonitoring area.

In addition, based on a change in video in response to the presence orabsence of product 70 in the set monitoring area, stock monitoringdevice 20 monitors a stock of product 70 in display shelf 50 andnotifies, for example, a stock manager of the monitoring result. Thus,it is possible to prevent a specific product from remaining lacking.Accordingly, it is possible to prevent a product selling opportunityfrom being lost, and thus, to provide customers with a positive image.

(Shelf Label Associating Process)

Between the detection of lacking (S14) and the output of information(S15), as illustrated by a dotted line in FIG. 6, shelf labelassociating section 215 may perform a shelf label associating process(S14 a).

The shelf label associating process illustrated as S14 a in FIG. 6 maybe performed as follows, for example. First, as illustrated in FIG. 9for example, shelf label associating section 215 sets, as a referenceshelf label, one (e.g., “third-row, first position” at the upper left)of shelf labels 51 detected through video recognition.

Then, based on shelf allocation information 400, as illustrated in FIG.10 for example, shelf label associating section 215 detects, forreference shelf label 51, shelf label 51 “having the closestx-coordinate among shelf labels 51 having close y-coordinates” asadjacent shelf label 51. In the example in FIG. 10, “third-row, secondposition” shelf label 51 is detected.

Shelf label associating section 215 repeats the process for detectingadjacent shelf label 51 illustrated in FIG. 10 until shelf label 51located at the end of the shelf row to which reference shelf label 51belongs, for example, “third-row, third position” shelf label 51, isdetected.

When shelf label 51 located at the end of the shelf row to whichreference shelf label 51 belongs is detected, shelf label associatingsection 215 detects, for shelf label 51 located at the end of the shelfrow to which reference shelf label 51 belongs, shelf label 51 “havingthe closest y-coordinate”. For example, as illustrated in FIG. 11,“second-row, third position” shelf label 51 is detected.

When shelf label 51 located at the end of the shelf row to whichreference shelf label 51 belongs is detected, as illustrated in FIG. 12,shelf label associating section 215 may set the search criterion toreference shelf label 51 again and may detect shelf label 51 “having theclosest y-coordinate”. In this case, “second-row, first position” shelflabel 51 is detected.

Shelf label associating section 215 repeats the shelf label detectionprocess illustrated in FIGS. 10 and 11 or FIGS. 10 and 12 until allshelf labels 51 are detected. FIG. 15A illustrates an example of asearch order in the shelf label detection process illustrated in FIGS.10 and 11, and FIG. 15B illustrates an example of a search order in theshelf label detection process illustrated in FIGS. 10 and 12. The orderfor searching for shelf label 51 is not limited to the ordersillustrated in FIGS. 15A and 15B.

When detection of all shelf labels 51 is completed, shelf labelassociating section 215 associates “m-th row, n-th position” shelf label51 (see FIG. 13) with shelf allocation information 400 (e.g., productinformation).

With the shelf label associating process, each shelf label 51 detectedthrough video recognition is associated with shelf allocationinformation 400 including the product information. Thus, even whenmonitoring area MA is set based on the distance between shelf labels 51(see FIG. 14), as described above, it is possible to determine theproduct information of a monitoring target in each monitoring area MA.The height of monitoring area MA in the top row may be set based on, forexample, the height of a lower shelf row.

Setting of reference shelf label 51 may be stored in storage 205, forexample. In and after a second-time shelf label associating process, thestored setting of reference shelf label 51 may be used. Thus, thereconfiguration of reference shelf label 51 may be unneeded.

In addition, shelf label associating section 215 may associate, forexample, either or both of information indicating a lack area detectedby lacking detector 213 and information of shelf label 51 detectedthrough video recognition with shelf allocation information 400.

(Generation of Notification Information Based on Shelf AllocationInformation)

In S15 in FIG. 6, output section 214 may generate notificationinformation based on shelf allocation information 400. For example,based on the association information between the information indicatinga lack area and the information of detected shelf label 51, outputsection 214 may generate notification information including a location(lack area) regarding detecting of lacking of product 70 and/or aproduct name and/or a product code of product 70 detected as lacking.The generated notification information is, for example, output to outputdevice 203 and/or communicator 206.

In this manner, by using shelf allocation information 400, stockmonitoring device 20 is available to notify, for example, a stockmanager of information on detecting of lacking including the productname and/or product code of individual product 70.

Thus, even when it is difficult to determine individual product name orthe like by video analysis of camera video, for example, the stockmanager is possible to accurately recognize which product 70 on whichshelf row is under lacking. Accordingly, the stock manager is availableto, for example, refill product 70 in individual shelf row accuratelyand smoothly.

(Example of Setting Monitoring Area Using Shelf Allocation Information)

Next, an example of setting monitoring area MA will be described. Inthis example, when shelf allocation information 400 includes, forexample, information indicating the size of product 70 and informationindicating the display number of product 70, monitoring area MA is setby using such information.

For example, shelf allocation information 400 includes informationindicating “three products 70 having a width X and a height Y aredisplayed for m-th row, n-th position shelf label 51”. In this case, asillustrated in FIG. 16, monitoring area setter 212 may set monitoringarea MA having such a size (width X and height Y) that includes threeproducts 70. Product 70 may have any size in the depth direction in theexample in FIG. 16.

As a result, monitoring area MA set in FIG. 16 and the monitoring areaset in FIG. 8 have the same size. However, in the example in FIG. 16,when product 70 has a different size and/or a different display number,the size of monitoring area MA to be set would be different from that inthe example in FIG. 8.

For example, when the size of product 70 in the example in FIG. 16 issmaller than that in the example in FIG. 8, or when the number ofdisplayed products 70 in the example in FIG. 16 is smaller than that inthe example in FIG. 8, the size of monitoring area MA to be set in theexample in FIG. 16 would be smaller than that in the example in FIG. 8.

All information elements of the width, height, and display number ofproduct 70 is not necessary to set monitoring area MA. As describedabove, the distance between shelf labels 51 is detected with videorecognition to set monitoring area MA. Thus, from the detected distance,monitoring area setter 212 is possible to compensate for a part ofinformation elements of the width, height, and display number of product70, even when any one of the information element is missing. Inaddition, the width and/or height of product 70 may be compensated forbased on the size of shelf label 51 detected through video recognition.The size of shelf label 51 may be a known size or may be detected withvideo recognition of shelf label 51.

Further, as in the top shelf row of the display shelf, even when shelflabel 51 as a reference to obtain the distance between shelf labels 51in the y-axis direction is not available, monitoring area setter 212 ispossible to appropriately set the size of monitoring area MA in they-axis direction based on information indicating the height of product70.

In addition, since the number of products 70, which are monitoringtargets in monitoring area MA, can be known, for example, lackingdetector 213 can more accurately perform stepwise detecting of lacking.Examples of the stepwise detecting of lacking will be described later.

(Example of Correcting Shelf Label Detection Result Using ShelfAllocation Information)

Next, an example of correcting a shelf label detection result by usingshelf allocation information 400 when shelf allocation information 400includes, for example, information indicating the type of product 70 andinformation indicating the display number of product 70 will bedescribed.

For example, when shelf allocation information 400 includes informationindicating “three types of products 70 having different widths (X) aredisplayed on shelf board 52 in the (m+1)-th row”, based on thisinformation, detected shelf label corrector 216 (see FIG. 5) can detectand correct failure in detecting shelf label 51 through videorecognition.

FIG. 17 illustrates a case where failure in detecting shelf label 51 mayoccur. In the example in FIG. 17, for shelf board 52 in the (m+1)-th(=second) row, three shelf labels 51 are supposed to be detected. Shelfallocation information 400 includes information indicating that, forthree shelf labels 51, respectively, products 70 represented as C1, C2,and C3 in FIG. 18 and having different widths (X1, X2, and X3) aredisplayed.

However, for example, another label (obstacle) “SALE!” partly overlapswith second-row, second-position shelf label 51. Thus, second-row,second-position shelf label 51 may be failed in the detection with videorecognition.

When the detection is failed, as illustrated in FIG. 19 for example,detected shelf label corrector 216 applies the widths (X1, X2, and X3)and display numbers of products 70 to shelf board 52 in the (m+1)-th(=second) row, thereby being available to estimate the presence of shelflabel 51 at the second row, second position. Detected shelf labelcorrector 216 may include estimated shelf label 51 in shelf labels 51detected by video recognition to correct the failure in detection.

In this manner, for example, even when there is shelf label 51 that mayfail to be detected by video recognition due to an obstacle, it ispossible to correct the failure in detection by image recognition basedon shelf allocation information 400. Thus, it is also possible to setmonitoring area MA with higher accuracy based on the position of shelflabel 51.

(Application of PTZ Camera)

By applying, for example, a PTZ camera as camera 10, and performingcontrol for changing the imaging direction of PTZ camera 10, it ispossible to capture multiple display shelves 50 or a plurality ofdifference display spaces of single display shelf 50 in the monitoringtarget.

FIG. 20 illustrates an example of applying PTZ camera 10. In the exampleillustrated in FIG. 20, two PTZ cameras 10 are arranged on, for example,a top of one of two display shelves 50 adjacent to each other in thedepth direction.

Thus, each PTZ camera 10 images, for example, the front of the other ofthe two display shelves 50 from obliquely above. PTZ camera 10, forexample, may also be arranged on the ceiling of a store where displayshelves 50 are arranged and may image the front of display shelves 50from obliquely above.

The imaging direction of each PTZ camera 10 may be changed in aplurality of directions (e.g., N=three) in the width direction ofdisplay shelves 50. “N” is an integer greater than or equal to 2denoting the number of imaging directions. Changing the imagingdirection may be controlled by, for example, stock monitoring device 20communicating with PTZ camera 10 via communication IF 261 (see FIG. 5).

Control for changing the imaging direction may be performed periodicallyin a preset cycle or aperiodically at a specific time. The imagingdirection may also be changed in accordance with an instruction from auser. For example, when a time slot during which specific product 70tends to be lacking is known in advance in a predetermined unit period,such as a day, a week, or a month, the imaging direction may becontrolled such that the monitoring target captures at least product 70that tends to be lacking during the time slot. A non-limiting example ofthe time slot during which specific product 70 tends to be lacking is aregular or irregular sales promotion time slot (sale time slot) ofspecific product 70.

Information indicating the imaging direction of PTZ camera 10 (e.g.,angle information) may be associated with shelf allocation information400 described above. For example, IDs may be assigned to PTZ camera 10and N (N is an integer greater than or equal to 2) angle informationitems, and the IDs may be associated with the IDs in shelf allocationinformation 400 (see FIG. 4).

Such association of the IDs enables, for example, identification ofdisplay shelf 50, the shelf row, PTZ camera 10, and the imagingdirection for monitoring. The IDs may be associated by, for example,shelf label associating section 215 (see FIG. 5).

In addition, as described above, the recognition model appropriate forvideo recognition changes depending on the imaging direction, and thus,the recognition model may be prepared in advance for each of differentimaging directions.

The recognition model appropriate for video recognition may also changedepending on whether, for example, a glass door is provided in the frontof display shelf 50. Thus, for example, the recognition modelappropriate for video recognition may be prepared in advance dependingon the imaging direction and/or whether a glass door is provided.

FIG. 21 illustrates an example in which recognition models #1, #2, and#3 are set for three imaging directions #A, #B, and #C, respectively.Alternatively, for example, a plurality of recognition models may beswitched so as to select a recognition model by which the number ofshelf labels 51 to be detected most closely matches the shelf allocationinformation.

In the above manner, by using PTZ camera 10 as camera 10, it is possibleto monitor and know about a stock at a plurality of locations withsingle PTZ camera 10. Thus, the total number of cameras to be arrangedin stock management system 1 can be reduced. This can reduce the totalcost of stock management system 1.

(Stepwise Detecting of Lacking)

Even when lack information is output after detection of a lack area ofproduct 70, it may not be possible to refill product 70 in time.

Thus, for example, in accordance with a ratio of the area of thebackground image appearing in the monitoring area set by monitoring areasetter 212, stock monitoring device 20 may provide the stock managerwith a stepwise stock level (i.e., “lack level”). FIGS. 22A to 22Cillustrate examples of stepwise lack levels.

FIG. 22A illustrates a condition (lack level 0) where three products 70are displayed at positions from “m-th row, n-th position” to “m-th row,(n+2)-th position”. FIG. 22B illustrates a condition (lack level 1)where one of three products 70 is lacking, and FIG. 22C illustrates acondition (lack level 3) where three products 70 are lacking.

Stock monitoring device 20 (e.g., lacking detector 213), for example,sets the lack level and the ratio of the area of the background imageappearing in the monitoring area at the lack level in association witheach other so as to determine the lack level relative to the ratio ofthe area of the background image appearing in the monitoring area.Information indicating the determined lack level is output to, forexample, output section 214.

Output section 214 generates notification information indicating theinformation indicating the lack level and outputs the notificationinformation to output device 203 and/or communicator 206. Thus, theinformation indicating the stepwise lack level is presented to, forexample, the stock manager.

Among the stepwise lack levels, the lack level at which the notificationis to be sent (i.e., notification threshold level) may be set to apredetermined level or may be set adaptively by, for example, the stockmanager. In the examples in FIGS. 22A to 22C, for example, “lack level1” may be set as the notification threshold level. The notification maybe provided for all lack levels.

In the above manner, when multiple products 70 are displayed in themonitoring area, stock monitoring device 20 is available to provide thestock manager with the stepwise lack level. Thus, at a stage beforeproducts 70 are completely lacking, the stock manager is available to benotified and to refill the product at an appropriate time.

When camera 10 is arranged at a position to image the shelf row fromobliquely above, stock monitoring device 20, as illustrated in FIGS. 23and 24 for example, may provide the stock manager with the lack level inthe depth direction (z-axis direction) of the shelf row. For example, itis possible to provide that frontmost products 70 are lacking.

When shelf allocation information 400 includes information indicatingthe display number of product 70 in the depth direction from the frontof the shelf row, based on a video recognition result of the monitoringarea and shelf allocation information 400, the lack level in the depthdirection of the shelf row may be set by, for example, lacking detector213.

(Mobile Object Removing Process)

A person (e.g., a customer) may pass through or may temporarily stop atthe front of display shelf 50. In such a case, as illustrated in FIG.25A for example, part or all of shelf label 51 may be hidden by theperson in camera video.

In such a case, the person in the camera video becomes a temporalobstacle for shelf label detection and sensing of lacking through videorecognition. The presence of the temporal obstacle may decrease theaccuracy of shelf label detection and detecting of lacking.

Thus, as an example of a pre-processing of video recognition, asillustrated in FIG. 25B, stock monitoring device 20 (e.g., shelf labeldetector 211 and/or lacking detector 213) may remove a mobile objectsuch as a person from the camera video.

When camera 10 supports a mobile object removing mode, for example,stock monitoring device 20 may communicate with camera 10 viacommunication IF 261 (see FIG. 5) to turn on the mobile object removingmode. A specific procedure for removing the mobile object is omittedfrom description because the aforementioned NPL 2 or the like is known.

Camera video of a few frames including the mobile object is subjected tothe mobile object removing process, and thus, the accuracy of shelflabel detection and detecting of lacking through video recognition canbe increased.

Effects of Embodiment 1

As described above, according to the above-described embodiment, basedon the position of shelf label 51 detected through video recognition ofcamera video including display shelf 50, the monitoring area is set fordisplay shelf 50 in the camera video, for example, without depending ona manual operation, the monitoring area can be appropriately set forcamera video including display shelf 50.

In other words, it is possible to set a display space as a monitoringtarget in display shelf 50 automatically. Accordingly, for example, evenwhen a relative position relationship between display shelf 50 andcamera 10 varies due to an external cause such as vibrations, detectionof shelf label 51 through video recognition enables resetting of anappropriate monitoring area. The effects of the above-describedembodiment are not limited to completely automatic setting of thedisplay spaces as a monitoring target. When a part of settings isperformed manually, such as manually designating a reference shelflabel, it is possible to increase the accuracy.

In addition, based on a change in video in response to the presence orabsence of product 70 in the set monitoring area, stock monitoringdevice 20 monitors a stock of product 70 in display shelf 50 andnotifies, for example, a stock manager of the monitoring result. Thus,it is possible to prevent a specific product from remaining lacking.This prevents a product selling opportunity from being lost, and as aresult, customers may have a positive image.

In addition, by using shelf allocation information 400, stock monitoringdevice 20 is available to notify, for example, stock manager of theproduct information of individual product 70 regarding detecting oflacking. Thus, for example, the stock manager is possible to recognizethe shelf row in which product 70 is lacking accurately. Even when it isdifficult to determine individual product name or the like by videoanalysis of camera video, the stock manager is available to, forexample, refill product 70 in individual shelf row accurately andsmoothly.

In addition, for example, even when there is shelf label 51 that mayfail to be detected through video recognition owing to an obstacle,stock monitoring device 20 is available to correct the failure indetection through image recognition based on shelf allocationinformation 400. Thus, monitoring area MA can be set with higheraccuracy based on the position of shelf label 51.

Furthermore, stock monitoring device 20 is available to, for example,determine a space or region occupied by multiple products 70 in a shelfrow with higher accuracy based on shelf allocation information 400.Accordingly, it is possible to increase the accuracy for setting themonitoring area based on the position of a shelf label, and thus, it isalso possible to increase the accuracy for detecting lacking of product70.

In addition, by using, for example, a PTZ camera as camera 10, it isunnecessary to arrange camera 10 for each display shelf 50 or eachdifferent region or space of display shelf 50, and thus, it is possibleto reduce the number of cameras 10 arranged in stock management system1.

This embodiment has described a PTZ camera as an example of camera 10.However, it is also possible to use a camera without a zooming functionwhen the camera is available to change the imaging direction. Inaddition, when there are restrictions on the arranging direction ofdisplay shelf 50, camera 10 does not need to have both a panningfunction and a tilting function.

For example, when display shelves 50 are available in substantially thehorizontal direction, even camera 10 with the tilting function in asingle direction is possible to capture multiple display shelves 50. Inaddition, camera 10 may be an omnidirectional camera.

When an omnidirectional camera is used as camera 10, an image obtainedby imaging is an omnidirectional image including video of multipledisplay shelves 50 around camera 10. In this case, control for changingthe imaging direction corresponds to a process for performing controlfor changing a portion to be extracted from the obtained omnidirectionalimage.

Stock monitoring device 20 performs shelf label detection and/ordetecting of lacking by using a recognition model appropriate for eachvirtual imaging direction corresponding to the portion extracted fromthe omnidirectional image, thereby performing substantially the samecontrol as that when using a PTZ camera.

The omnidirectional image is available to capture video of multipledisplay shelves 50. Thus, portions corresponding to respective displayshelves 50 can be extracted at the same time, and recognition modelsappropriate for the respective portions can be used for shelf labeldetection and/or detecting of lacking. Thus, multiple display shelves 50can be monitored at the same time.

In addition, by using, for example, the recognition model appropriatefor video recognition depending on the imaging direction and/or whethera glass door is provided of display shelf 50, stock monitoring device 20is possible to increase the accuracy for shelf label detection and/orlacking detection.

In addition, in a case where multiple products 70 are displayed in themonitoring area, stock monitoring device 20 is available to provide thestock manager with the stepwise lack level. Thus, at a stage beforeproducts 70 are completely lacking, stock monitoring device 20 isavailable to notify reminder information to the stock manager.Therefore, the stock manager can refill the product at an appropriatetime.

Furthermore, camera video of a few frames including the mobile object issubjected to the mobile object removing process, and thus, the accuracyof shelf label detection and detecting of lacking through videorecognition can be increased in stock monitoring device 20.

(Others)

Although the above embodiment has described an example in which themonitoring target of computer 20 is product 70 displayed in displayshelf 50, the monitoring target of computer 20 is not limited to product70 for commercial transaction. For example, the monitoring target ofcomputer 20 may be “object” such as an exhibit displayed in a showcase.In this case, for example, computer 20 is possible to detect missing of“object” from a showcase and notify a manager or the like.

Computer 20 that monitors a stock or display condition of “article” maybe referred to as, for example, “shelf monitoring device”, “articlemonitoring device”, “article display condition monitoring device”, orthe like. In addition, a program causing computer 20 to function as adevice that monitors a stock or display condition of “article” may bereferred to as “shelf monitoring program”, “article monitoring program”,“article display condition monitoring program”, or the like.

Embodiment 2

Embodiment 2 will describe a shelf label detection device that increasesthe accuracy of the shelf label position. Description of part common toEmbodiment 1 will be omitted and part different from Embodiment 1 willbe described in Embodiment 2.

(Outline)

FIGS. 26A to 26D illustrate issues in a case where the shelf labelposition is determined only by a known video recognition technique.

In a case where the shelf label position is determined by a known videorecognition technique, exemplarily, from camera video of display shelf50, a shelf label image is recognized by pattern matching, and the shelflabel position is determined.

However, in the recognition of the shelf label image by a videorecognition technique, erroneous detection and/or failure in detectionmay occur in the following (A1) to (A3) cases, for example.

-   (A1) As illustrated in FIG. 26A, shelf label image 301 has a low    resolution.-   (A2) As illustrated in FIG. 26B, shelf label image 302 is covered    with an obstacle, such as a display product.-   (A3) As illustrated in FIG. 26C, object image 304 similar to shelf    label image 303 is present.

For example, as illustrated in FIG. 26D, when camera video includesshelf label images 301 and 302 in (A1) and (A2) above, although fourshelf label images 301, 302, 305, and 306 are present, only two shelflabel images 305 and 306 are detected only by a video recognitiontechnique, and the other two shelf label images 301 and 302 may fail tobe detected.

FIGS. 27A and 27B illustrate issues in a case where the shelf labelposition is manually input.

For example, as illustrated in FIG. 27A, when a user inputs the positionof a shelf label image in camera video, it is possible to preventerroneous detection or failure in detection of the shelf label image inthe video recognition technique.

However, when the shelf label position is manually determined, in thefollowing (B1) and (B2) cases for example, as illustrated in FIG. 27B,the shelf label position having been input by the user and the positionof the shelf label image in the camera video may be misaligned eachother.

-   (B1) Case where the camera video is changed by panning or tilting.-   (B2) Case where the camera video is changed by camera 10 receiving a    shock or the like.

When the user is made to input the position of the shelf label againeach time the camera video is changed as set forth above, the user issignificantly burdened. In (B1), it is difficult to cancel themisalignment by restoring the panning or tilting angle to that beforechange because it is physically difficult to set the completelyidentical angle again, and, when the distance between the camera and thesubject is long, a slight shift impacts a significant shift in thevideo. Accordingly, when the panning or tilting angle is restored to theoriginal in (B1), the misalignment in the position of the shelf labelimage needs to be corrected by some kinds of solutions.

FIG. 28 illustrates an outline of a method for determining shelf labelpositions according to this embodiment. Next, the method for determiningshelf label positions according to this embodiment in order to solve theabove issues will be described with reference to FIG. 28.

As illustrated in FIG. 28, in the method for determining shelf labelpositions according to this embodiment, manual shelf label position set503 is corrected by using automatic shelf label position set 504 togenerate corrected shelf label position set 505. Manual shelf labelposition set 503 includes shelf label positions that are input inadvance by a user in reference camera video 501. Automatic shelf labelposition set 504 includes shelf label positions detected from monitoringcamera video 502 with a video recognition technique. Subsequently, byusing corrected shelf label position set 505, the shelf label positionsin monitoring camera video 502 are determined.

The shelf label position may be information indicating the position of ashelf label image in camera video (e.g., XY coordinate point). Inaddition, corrected shelf label position set 505 may be obtained bymoving positions in manual shelf label position set 503 such that asmany shelf label positions included in manual shelf label position set503 approach as many shelf label positions included in automatic shelflabel position set 504 in monitoring camera video 502 as possible.

The shelf label position may also include, in addition to theinformation indicating the position of a shelf label image, informationindicating a region of the shelf label image (hereinafter referred to as“shelf label region”). In addition, corrected shelf label position set505 may be obtained by moving positions in manual shelf label positionset 503 such that a shelf label region included in manual shelf labelposition set 503 most largely overlaps with a shelf label regionincluded in automatic shelf label position set 504.

Thus, even when camera video is changed to a certain extent, it isavailable to determine the shelf label positions from the camera videowith high accuracy without manually inputting the positions of shelflabel images by a user again.

Hereinafter, details of this embodiment will be described. The methodfor determining shelf label positions according to this embodiment maybe included in stock monitoring device 20 described in Embodiment 1.

(Configuration of Shelf Label Detection Device)

FIG. 29 is a block diagram illustrating a configuration example of shelflabel detection device 2 according to this embodiment. As illustrated inFIG. 29, shelf label detection device 2 includes video obtainer 401,manual shelf label setter 402, automatic shelf label detector 403, shelflabel position corrector 404, and shelf label position determiner 405.

Video obtainer 401 obtains camera video from camera 10 at apredetermined time (e.g., in a predetermined cycle) and stores thecamera video in one or both of memory 204 and storage 205 (hereinafterreferred to as “memory or the like”). Herein, camera video used bymanual shelf label setter 402 is referred to as reference camera video501, and camera video used by automatic shelf label detector 403 isreferred to as monitoring camera video 502.

Manual shelf label setter 402 obtains reference camera video 501 fromthe memory or the like, displays reference camera video 501 on outputdevice 203 (e.g., a display device), and receives input of the positionof a shelf label image (shelf label position) in reference camera video501 from a user through input device 202. In addition, manual shelflabel setter 402 generates manual shelf label position set 503 includinginformation of each shelf label position that is input and stores manualshelf label position set 503 in the memory or the like.

Automatic shelf label detector 403 obtains monitoring camera video 502from the memory or the like, detects each shelf label image frommonitoring camera video 502 with a video recognition technique, anddetermines each shelf label position. In addition, automatic shelf labeldetector 403 generates automatic shelf label position set 504 includinginformation of each shelf label position that is determined and storesautomatic shelf label position set 504 in the memory or the like.

Shelf label position corrector 404 corrects manual shelf label positionset 503 to match monitoring camera video 502 by using automatic shelflabel position set 504 and generates corrected shelf label position set505. In addition, shelf label position corrector 404 stores generatedcorrected shelf label position set 505 in the memory or the like. Forexample, shelf label position corrector 404 calculates an offset amount(movement amount) of manual shelf label position set 503 with which aplurality of shelf label regions included in manual shelf label positionset 503 and a plurality of shelf label regions included in automaticshelf label position set 504 have the maximum overlapping area. Shelflabel position corrector 404 moves manual shelf label position set 503by the calculated offset amount to generate corrected shelf labelposition set 505. Details of the process of shelf label positioncorrector 404 will be described later.

By using corrected shelf label position set 505, shelf label positiondeterminer 405 determines each shelf label position in monitoring cameravideo 502. The shelf label position determined by shelf label positiondeterminer 405 may be used by, for example, shelf label detector 211 ofstock monitoring device 20 in FIG. 5.

As illustrated in S13 a in the flowchart of FIG. 30, each shelf labelposition determined by shelf label position determiner 405 may be inputand used in the display condition detection process in S14 in theflowchart illustrated in FIG. 6. This increases the accuracy of theshelf label position and also the accuracy of the display conditiondetection.

<Details of Manual Shelf Label Setter>

Next, details of manual shelf label setter 402 will be described withreference to the flowchart in FIG. 31.

Manual shelf label setter 402 displays reference camera video 501 (S101)and receives input of a shelf label position from a user (S102).

Through input device 202 for example, the user inputs the position ofeach shelf label image included in displayed reference camera video 501(S103).

From the position of each shelf label image input in S103, manual shelflabel setter 402 generates information of the shelf label positioncorresponding to each shelf label image. From the generated shelf labelpositions, manual shelf label setter 402 generates manual shelf labelposition set 503 and stores manual shelf label position set 503 in thememory or the like (S104).

Though the above process, manual shelf label setter 402 generates manualshelf label position set 503 and stores manual shelf label position set503 in the memory or the like.

(Details of Shelf Label Position Corrector)

Next, details of shelf label position corrector 404 will be describedwith reference to the flowchart in FIG. 32. This process may beperformed each time when monitoring camera video 502 is updated.

Based on manual shelf label position set 503, shelf label positioncorrector 404 sets a plurality of offset amounts that are different fromeach other (S201). Details of setting the plurality of offset amountswill be described later.

Subsequently, shelf label position corrector 404 repeats an offset loopprocess (S202 to S205) for the number of the plurality of offset amountsthat are set in S201 (S202). At this time, in each offset loop process,shelf label position corrector 404 selects a different offset amountfrom the plurality of offset amounts that are set in S201. In thedescription of the offset loop process, the selected offset amount isreferred to as “selected offset amount”.

Subsequently, shelf label position corrector 404 moves the positions inmanual shelf label position set 503 by the selected offset amount. Shelflabel position corrector 404 calculates an overlapping area betweenshelf label regions in moved manual shelf label position set 503 andshelf label regions in automatic shelf label position set 504 inmonitoring camera video 502 (S203).

Subsequently, shelf label position corrector 404 stores the selectedoffset amount and the overlapping area calculated in S203 in associationwith each other in the memory or the like (S204).

After completion of the offset loop process (S205), shelf label positioncorrector 404 determines the offset amount associated with the largestoverlapping area by referring to the memory or the like (S206).

Subsequently, shelf label position corrector 404 moves manual shelflabel position set 503 by the offset amount determined in S206,generates corrected shelf label position set 505 that matches monitoringcamera video 502, and stores corrected shelf label position set 505 inthe memory or the like (S207).

Next, details of the process in S201 in FIG. 32 will be described withreference to FIG. 33.

First, shelf label position corrector 404 calculates the average sizeand shape of shelf label regions (hereinafter referred to as “averageshelf label region”) from the sizes and shapes of the shelf labelregions included in manual shelf label position set 503. The shape ofthe average shelf label region may be rectangular as in oblique-lineregion 601 in FIG. 33.

Subsequently, as illustrated in FIG. 33, shelf label position corrector404 sets a plurality of offset amounts that are different from eachother by setting, as a unit amount of the offset amount, Δx and Δy whereΔx is half the length of the average shelf label region in the x-axisdirection and Δy is half the length of the average shelf label region inthe y-axis direction. For example, FIG. 33 illustrates a case where 25offset amounts are set in total as −2Δx, −Δx, 0, +Δx, and +2Δx in thex-axis direction and −2Δy, −Δy, 0, +Δ, and +2Δy in the y-axis direction.In a case where 25 offset amounts are set in this manner, the repeatnumber of the offset loop process (S202) in FIG. 32 is 25.

In addition, for example, when the selected offset amount in the offsetloop process corresponds to shaded region 602 in FIG. 33 (−Δx, +Δy), inS203, shelf label position corrector 404 moves each shelf label regionin manual shelf label position set 503 by the offset amount (−Δx, +Δy)and calculates the overlapping area. Point 603 in FIG. 33 indicates ashelf label position (coordinate point) in a case where shelf labelregion 601 is moved by the offset amount. The shelf label position maybe the center (or the center of gravity) coordinates of a shelf labelregion.

Through the above process, shelf label position corrector 404 calculatesthe offset amount with which manual shelf label position set 503 andautomatic shelf label position set 504 most largely overlap with eachother, and generates corrected shelf label position set 505 by using thecalculated offset amount.

The shape of the average shelf label region illustrated in FIG. 33 is anexample. As illustrated in FIGS. 26A to 26D for example, when the shapeof each shelf label image in camera video is distorted by theperspective, the shape of the average shelf label region may be similarto the shape of the distorted shelf label image.

In addition, the above “overlapping area” may be replaced with“overlapping ratio” indicating the ratio of overlap between the shelflabel regions in manual shelf label position set 503 and the shelf labelregions in automatic shelf label position set 504.

When the overlapping area is less than a predetermined threshold, shelflabel position corrector 404 may issue an alert indicating that theoverlapping area is extremely small. This condition may occur typicallywhen the environment for arranging camera 10 changes significantly. Thealert can quickly notify a user of a large change in the environment forarranging camera 10.

In addition, although the repeat number of the offset loop process(S202) is 25 in this embodiment, the number of times may be larger orsmaller. For example, when the number of times is increased, the offsetamount is changed in a larger number of levels; when the number of timesis decreased, the offset amount is changed in a smaller number oflevels. Herein, the level of change in the offset amount and the timefor process are in a trade-off relationship. That is, a larger number oflevels for the change in the offset amount increases the accuracy, and asmaller number of levels reduces the time for process.

In addition, in this embodiment, part of the process for correcting theshelf label position may be performed by input from a user or the like.Examples of the input from a user include input for assisting processingin the offset loop process (S202), such as setting the offset maximumvalue, setting an offset change amount, and specifying the position of ashelf label (e.g., an upper left shelf label). In addition, for example,a plurality of offset amounts and an overlapping amount between manualshelf label position set 503 and automatic shelf label position set 504in each case may be output so that a user may determine the offsetamount to be used.

In addition, corrected shelf label position set 505 in this embodimentmay be generated when the camera performs a panning operation, a tiltingoperation, or the like, or may be generated in a predetermined cycle. Inthe former case, the number of times of correction can be reduced. Inthe latter case, correction can be performed relatively quickly also ina case where the camera is moved by an unintended impulse or the like.

Effects of Embodiment 2

In this embodiment, shelf label detection device 2 corrects manual shelflabel position set 503 generated for reference camera video 501 by usingautomatic shelf label position set 504 generated for monitoring cameravideo 502 to generate corrected shelf label position set 505 thatmatches monitoring camera video 502. Since corrected shelf labelposition set 505 is obtained by correcting manual shelf label positionset 503 with high accuracy to be applied to monitoring camera video 502,shelf label positions in monitoring camera video 502 can be detectedwith high accuracy.

Embodiment 3

Shelf label detection device 2 according to Embodiment 3 is differentfrom shelf label detection device 2 according to Embodiment 2 inincluding various notification functions. Description of part common toEmbodiment 2 will be omitted and part different from Embodiment 2 willbe described in Embodiment 3.

FIG. 34 is a block diagram illustrating a configuration example of shelflabel detection device 2 according to this embodiment. As illustrated inFIG. 34, shelf label detection device 2 includes, in addition to theconfiguration of shelf label detection device 2 illustrated in FIG. 29,change detector 406 and modifier 407.

Change detector 406 detects a change in arrangement of shelf labels 51in display shelf 50. In addition, change detector 406 issues an alert inaccordance with the detected content. Change detector 406 may also bereferred to as “notifying section”.

When, for example, change detector 406 detects misalignment betweenmanual shelf label position set 503 and automatic shelf label positionset 504 over a predetermined period or longer, change detector 406issues a first update request alert regarding manual shelf labelposition set 503. The misalignment exemplarily occurs when thearrangement of shelf labels 51 in display shelf 50 is changed but manualshelf label position set 503 is not changed. Herein, the condition of“predetermined period or longer” prevents an alert from being issuederroneously when misalignment is temporarily detected by erroneousrecognition or the like by a video recognition technique or the likedespite of no change in the arrangement of shelf labels 51. Details of aprocess for issuing the first update request alert will be describedlater.

In addition, when, for example, change detector 406 detects misalignmentbetween shelf allocation information 400 and corrected shelf labelposition set 505, change detector 406 issues a second update requestalert regarding manual shelf label position set 503. The misalignmentexemplarily occurs when, after the arrangement of shelf labels 51 indisplay shelf 50 has been changed, shelf allocation information 400 isupdated but manual shelf label position set 503 is not updated, or whenmanual shelf label position set 503 is updated but shelf allocationinformation 400 is not updated. Details of a process for issuing thesecond update request alert will be described later.

When modifier 407 detects the misalignment between manual shelf labelposition set 503 and automatic shelf label position set 504 over apredetermined period or longer, modifier 407 automatically modifiesmanual shelf label position set 503. The misalignment exemplarily occurswhen, after the arrangement of shelf labels 51 in display shelf 50 hasbeen changed, manual shelf label position set 503 is not changed. Forexample, modifier 407 modifies manual shelf label position set 503 bydeleting, from manual shelf label position set 503, a shelf labelposition that is present in corrected shelf label position set 505 butis not present in automatic shelf label position set 504 and by adding,to manual shelf label position set 503, a shelf label position that ispresent in automatic shelf label position set 504 but is not present incorrected shelf label position set 505. Details of the process ofmodifier 407 will be described later.

(Details of Issuing First Update Request Alert)

Next, a specific example of a process for issuing the first updaterequest alert will be described with reference to FIG. 35.

In FIG. 35, change detector 406 detects manual shelf label positions701, 702, and 703 that do not overlap with automatic shelf labelposition set 504 (hereinafter referred to as “non-overlapping manualshelf labels”) in a first-time process, detects non-overlapping manualshelf labels 702 and 703 in a second-time process, and detectsnon-overlapping shelf label 703 in a third-time process. The term“non-overlapping” is not limited to a case where shelf label regions inmanual shelf label position set 503 and shelf label regions in automaticshelf label position set 504 do not overlap with each other at all. Forexample, in order to determine non-overlapping, change detector 406 mayprovide a threshold for the overlapping ratio in advance and maydetermine non-overlapping when the overlapping ratio is less than thethreshold. In FIG. 35, the number of times of detection ofnon-overlapping manual shelf label 703 reaches “3” in the third-timeprocess. Herein, when the threshold of the number of times of detectionfor issuing an alert is “3”, since non-overlapping manual shelf label703 is detected continuously for three times (i.e., over a predeterminedperiod or longer), change detector 406 issues the first update requestalert.

Next, details of the process for issuing the first update request alertwill be described with reference to the flowchart in FIG. 36. Theprocess may be performed after S13 a in FIG. 30.

Change detector 406 compares manual shelf label position set 503 andautomatic shelf label position set 504 with each other to perform aprocess for detecting a manual shelf label position not overlapping withautomatic shelf label position set 504 (i.e., “non-overlapping manualshelf label”) (S301).

Subsequently, the change detector determines whether the non-overlappingmanual shelf label has been detected in S301 (S302).

When the non-overlapping manual shelf label has not been detected (S302:NO), change detector 406 ends this process.

When the non-overlapping manual shelf label has been detected (S302:YES), change detector 406 repeats a change detection loop process (S303to S310) for the number of detected non-overlapping manual shelf labels.At this time, in each change detection loop process, change detector 406selects a different non-overlapping manual shelf label. In thedescription of FIG. 36, the selected non-overlapping manual shelf labelis referred to as “selected non-overlapping manual shelf label”.

Change detector 406 determines whether the selected non-overlappingmanual shelf label is registered in an exceptional shelf label table(not illustrated) (S304). Herein, in the exceptional shelf label table,manual shelf label positions corresponding to shelf labels 51 thatcannot be recognized by a video recognition technique owing to, forexample, an obstacle or the like are registered.

When the selected non-overlapping manual shelf label is registered inthe exceptional shelf label table (S304: YES), change detector 406proceeds to the next-time change detection loop process (S310).

When the selected non-overlapping shelf label is not registered in theexceptional shelf label table (S304: NO), change detector 406 determineswhether the selected non-overlapping manual shelf label is registered ina shelf-label-of-interest table (S305). Herein, in theshelf-label-of-interest table, the non-overlapping manual shelf labeldetected in the previous S301 is registered.

When the selected non-overlapping manual shelf label is not registeredin the shelf-label-of-interest table (not illustrated) (S305: NO),change detector 406 registers the selected non-overlapping manual shelflabel in the shelf-label-of-interest table (S306) and proceeds to S307.

When the selected non-overlapping manual shelf label is registered inthe shelf-label-of-interest table (S305: YES), change detector 406proceeds to S307.

In S307, change detector 406 increments the number of times of detectionof the selected non-overlapping manual shelf label by “1” (S307).

Subsequently, change detector 406 determines whether the number of timesof detection of the selected non-overlapping manual shelf label isgreater than or equal to a predetermined threshold (S308).

When the number of times of detection of the selected non-overlappingmanual shelf label is less than the predetermined threshold (S308: NO),change detector 406 proceeds to the next-time change detection loopprocess (S310).

When the number of times of detection of the selected non-overlappingmanual shelf label is greater than or equal to the predeterminedthreshold (S308: YES), change detector 406 issues the first updaterequest alert (S309) and proceeds to the next-time change detection loopprocess (S310). At this time, change detector 406 may also issue anotification indicating that a misaligned target is the selectednon-overlapping manual shelf label.

After completion of the change detection loop process (S310), changedetector 406 deletes the non-overlapping manual shelf label that is notdetected in the current S301 from the shelf-label-of-interest table(S311). Thus, the non-overlapping manual shelf label that has beencontinuously detected remains in the shelf-label-of-interest table.Subsequently, change detector 406 ends this process.

Through the above process, when change detector 406 detects thenon-overlapping manual shelf label continuously for a predeterminednumber of times or more (i.e., over a predetermined period or longer),change detector 406 issues the first update request alert. This alertcan notify a user that manual shelf label position set 503 needs to beupdated.

The condition for issuing the first update request alert is not limitedto the condition that the same non-overlapping manual shelf label iscontinuously detected as above. For example, the condition for issuingthe first update request alert may be a condition that the samenon-overlapping manual shelf label is detected at a frequency that ishigher than or equal to a predetermined threshold.

(Details of Issuing Second Update Request Alert)

Next, a specific example of a process for issuing the second updaterequest alert will be described with reference to FIG. 37.

FIG. 37 illustrates that the number of shelf labels 51 arranged on shelfboard 52 in the first row is changed from “6” to “5” and the number ofshelf labels 51 arranged on shelf board 52 in the third row is changedfrom “6” to “7” in shelf allocation information 400, but manual shelflabel position set 503 is not changed (the number of shelf labels 51 onboth shelf boards 52 remains “6”). In this case, since misalignmentoccurs between shelf allocation information 400 and manual shelf labelposition set 503 on shelf board 52 in the first row and shelf board 52in the third row, change detector 406 issues the second update requestalert. At this time, change detector 406 may also issue a notificationabout the number of shelf boards 52 where misalignment occurs.

Next, details of the process for issuing the second update request alertwill be described with reference to the flowchart in FIG. 38. Theprocess may be performed after S13 a in FIG. 30.

Change detector 406 determines the number of shelf boards 52 in displayshelf 50 that is a monitoring target from shelf allocation information400 (S401).

Change detector 406 repeats a shelf board loop process (S402 to S405)for the number of shelf boards 52 determined in S401 (S402). Herein, ineach shelf board loop process, change detector 406 selects differentshelf board 52. In the description of the shelf board loop process,selected shelf board 52 will be referred to as “selected board”.

Change detector 406 compares the number of shelf labels 51 on theselected board in shelf allocation information 400 and the number ofshelf labels 51 on the selected board in manual shelf label position set503 with each other and determines whether there is a discrepancybetween the two numbers of shelf labels (S403).

When there is no discrepancy between the two numbers of shelf labels(S403: NO), change detector 406 proceeds to the next-time shelf boardloop process (S405).

When there is a discrepancy between the two numbers of shelf labels(S403: YES), change detector 406 issues the second update request alert(S404) and proceeds to the next-time shelf board loop process (S405).

After the completion of the shelf board loop process (S405), changedetector 406 ends this process.

Through the above process, when misalignment occurs between shelfallocation information 400 and manual shelf label position set 503,change detector 406 issues the second update request alert. This alertcan notify a user that manual shelf label position set 503 or shelfallocation information 400 needs to be updated.

The condition for issuing the second update request alert is themisalignment between shelf allocation information 400 and the manualshelf label position set 503 in the above description. However, theissuing condition may be misalignment between shelf allocationinformation 400 and corrected shelf label position set 505. In thiscase, since corrected shelf label position set 505 is obtained bycorrecting manual shelf label position set 503 by using the informationof automatic shelf label position set 504, even when the video shifts,the number of shelf labels 51 arranged on shelf board 52 can beaccurately determined.

In addition, the presence or absence of misalignment is determined fromthe number of shelf labels 51 arranged on each shelf board 52 in theabove description. However, the presence or absence of misalignment maybe determined based on other conditions. For example, it may bedetermined whether the number of shelf labels in each column of theshelf matches shelf allocation information 400, or it may be determined,when shelf allocation information 400 includes information such as theinterval between shelf labels 51, whether the interval between shelflabels 51 matches shelf allocation information 400.

(Details of Automatic Modification of Manual Shelf Label Position Set)

Next, a specific example of a process for automatically modifying manualshelf label position set 503 will be described with reference to FIG.39.

In FIG. 39, in a change on certain shelf board 52, shelf label 801 atthe second position from the left is removed, and shelf label 802 isattached between the fifth position and the sixth position from theleft. In this case, when manual shelf label position set 503 is notchanged, misalignment occurs between corrected shelf label position set505 and the shelf label image as illustrated as the first to third timesin FIG. 39.

Thus, in the first-time to third-time processes, modifier 407 detectsunknown shelf label 803 that is present in corrected shelf labelposition set 505 but not present in automatic shelf label position set504 and unknown shelf label 804 that is present in automatic shelf labelposition set 504 but not present in corrected shelf label position set505. Herein, when the threshold of the number of times of detection forautomatic modification is “3”, since unknown shelf labels 803 and 804are continuously detected for three times, modifier 407 deletes unknownshelf label 803 (805) from manual shelf label position set 503 and addsunknown shelf label 804 (806) to manual shelf label position set 503.Through this process, as illustrated in the fourth-time process in FIG.39, misalignment between corrected shelf label position set 505 and theshelf label image is solved.

The process for automatic modification of manual shelf label positionset 503 will be described with reference to the flowchart in FIG. 40.This process may be performed after S13 a in FIG. 30.

Modifier 407 compares corrected manual shelf label position set 503 andautomatic shelf label position set 504 with each other and performs aprocess for detecting a shelf label position (i.e., unknown shelf label)that is present in one of the sets but is not present in the other(S501).

Next, modifier 407 determines whether the unknown shelf label has beendetected in S501 (S502).

When the unknown shelf label has not been detected (S502: NO), modifier407 ends this process.

When the unknown shelf label has been detected (S502: YES), modifier 407determines whether all of unknown shelf labels that have been detectedare registered in an unknown shelf label table (not illustrated) (S503).Herein, in the unknown shelf label table, the unknown shelf labeldetected in the previous S501 is registered.

When at least one of the unknown shelf labels that have been detected isnot registered in the unknown shelf label table (S503: NO), modifier 407registers the unregistered unknown shelf label in the unknown shelflabel table (S504) and proceeds to S505.

When all the unknown shelf labels that have been detected are registeredin the unknown shelf label table (S503: YES), modifier 407 proceeds toS505.

In S505, modifier 407 increments the number of times of detection ofeach unknown shelf label registered in the unknown shelf label table by“1” (S505).

Subsequently, modifier 407 determines whether the unknown shelf labeltable includes an unknown shelf label for which the number of times ofdetection is greater than or equal to a predetermined threshold (S506).

When the unknown shelf label table does not include an unknown shelflabel for which the number of times of detection is greater than orequal to the predetermined threshold (S506: NO), modifier 407 proceedsto S508.

When the unknown shelf label table includes an unknown shelf label forwhich the number of times of detection is greater than or equal to thepredetermined threshold (S506: YES), modifier 407 modifies manual shelflabel position set 503 by using the unknown shelf label for which thenumber of times of detection is greater than or equal to the threshold(S507) and proceeds to S508. Specifically, modifier 407 deletes, frommanual shelf label position set 503, an unknown shelf label that ispresent in corrected shelf label position set 505 but is not present inautomatic shelf label position set 504 and for which the number of timesof detection is greater than or equal to the threshold. In addition,modifier 407 adds, to manual shelf label position set 503, an unknownshelf label that is present in automatic shelf label position set 504but is not present in corrected shelf label position set 505 and forwhich the number of times of detection is greater than or equal to thevalue. At this time, modifier 407 may issue an alert indicatingmodification of manual shelf label position set 503 and may modifymanual shelf label position set 503 only when a user allows.

Subsequently, modifier 407 deletes, from the unknown shelf label table,an unknown shelf label that has not been detected in S501 (S508). Thus,unknown shelf labels that have been continuously detected remains in theunknown shelf label table. Subsequently, modifier 407 ends this process.

Through the above process, when modifier 407 detects an unknown shelflabel continuously for a predetermined number of times or more (i.e.,over a predetermined period or longer), modifier 407 automaticallymodifies manual shelf label position set 503. This automaticmodification reduces a process performed by a user for manually updatingmanual shelf label position set 503 after changing the arrangement ofshelf labels.

The condition for issuing the second update request alert is not limitedto the condition that an unknown shelf label is continuously detected asabove. For example, the condition for issuing the second update requestalert may be a condition that the same unknown shelf label is detectedat a frequency that is higher than or equal to a predeterminedthreshold.

In addition, in this embodiment, the threshold for a condition forautomatically adding unknown shelf label 804 to manual shelf labelposition set 503 and the threshold for a condition for automaticallydeleting unknown shelf label 803 from manual shelf label position set503 may be set to different values. For example, the threshold for theautomatically adding condition may be set to be higher than thethreshold for the automatically deleting condition. Because, inautomatic shelf label position set 504, a non-existing shelf label islikely to be erroneously detected owing to limited accuracy of imagerecognition.

Furthermore, when the number of unknown shelf labels 803 deleted bymodifier 407 exceeds a predetermined number or when the ratio of thenumber of deleted unknown shelf labels 803 to all shelf labels exceeds apredetermined ratio, for example, a further alert may be issued.Because, when a large number of shelf labels are automatically modified,information of manual shelf label position set 503 may have become oldwith high possibility. By issuing such an alert, a user can be promptedto review manual shelf label position set 503.

Effects of Embodiment 3

In this embodiment, when a misalignment between manual shelf labelposition set 503 and automatic shelf label position set 504 is detectedover a predetermined period or longer, shelf label detection device 2issues the first update request alert regarding manual shelf labelposition set 503. In addition, when a misalignment between shelfallocation information 400 and corrected shelf label position set 505 isdetected, shelf label detection device 2 may issue the second updaterequest alert regarding manual shelf label position set 503. Such alertscan prevent a user from forgetting to update manual shelf label positionset 503 after changing arrangement of shelf labels 51.

Furthermore, when a misalignment between manual shelf label position set503 and automatic shelf label position set 504 is detected over apredetermined period or longer, shelf label detection device 2automatically modifies manual shelf label position set 503. Thisautomatic modification can save a user manually updating manual shelflabel position set 503 after changing arrangement of shelf labels 51.

Some embodiments of the present disclosure have been described above.

The functional blocks used in the above description of the embodimentsare typically implemented as an LSI, which is an integrated circuit.Each of the functional blocks may be implemented as a single chip, orsome or all of the functional blocks may be integrated into a singlechip. Although the integrated circuit is herein called an LSI, theintegrated circuit may be called an IC, a system LSI, a super LSI, or anultra LSI depending on the difference in the degree of integration.

In addition, the technique for circuit integration is not limited toLSI, and circuit integration may be implemented by using a dedicatedcircuit or a general-purpose processor. An FPGA (Field Programmable GateArray) that is programmable after manufacturing the LSI, or areconfigurable processor for which connections and settings of circuitcells within the LSI can be reconfigured may be used.

Furthermore, in a case where a technique for circuit integration thatreplaces LSI emerges with the advancement of semiconductor technology orbased on any technology that is separately derived, the functionalblocks may be integrated by using the technique, as a matter of course.Application of, for example, biotechnology is possible.

Summary of Present Disclosure

A shelf label detection device according to the present disclosureincludes: an obtainer that obtains first camera video and second cameravideo including an image of a shelf label arranged in a display shelf; acorrector that corrects a first shelf label position set to generate acorrected shelf label position set by using a second shelf labelposition set, the first shelf label position set including a shelf labelposition that is set in advance in the first camera video, the secondshelf label position set including a shelf label position for the imageof the shelf label detected from the second camera video through videorecognition; and an determiner that determines the shelf label positionin the second camera video by using the corrected shelf label positionset.

In the shelf label detection device according to the present disclosure,the corrector may calculate a movement amount of the first shelf labelposition set based on a size of an overlapping amount between a regionof the shelf label position included in the first shelf label positionset and a region of the shelf label position included in the secondshelf label position set, and generate the corrected shelf labelposition set by using the movement amount.

The shelf label detection device according to the present disclosure mayfurther include: a first notification section that sends a predeterminedfirst notification when the first shelf label position set and thesecond shelf label position set are misaligned over a predeterminedperiod or longer.

The shelf label detection device according to the present disclosure mayfurther include: a second notification section that sends apredetermined second notification when shelf allocation information,which is information on an arrangement position of the shelf label inthe display shelf, is misaligned with the corrected shelf label positionset.

The shelf label detection device according to the present disclosure mayfurther include: a modifier that modifies the first shelf label positionset by using the second shelf label position set when a shelf labelposition that is present in any one of the corrected shelf labelposition set and the second shelf label position set is detected over apredetermined period or longer.

A shelf label detection method according to the present disclosureincludes: obtaining first camera video and second camera video includingan image of a shelf label arranged in a display shelf; correcting afirst shelf label position set to generate a corrected shelf labelposition set by using a second shelf label position set, the first shelflabel position set including a shelf label position that is set inadvance in the first camera video, the second shelf label position setincluding a shelf label position for the image of the shelf labeldetected from the second camera video through video recognition; anddetermining the shelf label position in the second camera video by usingthe corrected shelf label position set.

A shelf label detection program according to the present disclosurecauses a computer to execute a process including: obtaining first cameravideo and second camera video including an image of a shelf labelarranged in a display shelf; correcting a first shelf label position setto generate a corrected shelf label position set by using a second shelflabel position set, the first shelf label position set including a shelflabel position that is set in advance in the first camera video, thesecond shelf label position set including a shelf label position for theimage of the shelf label detected from the second camera video throughvideo recognition; and determining the shelf label position in thesecond camera video by using the corrected shelf label position set.

This patent application claims priority based on Japanese PatentApplication No. 2017-209447 filed on Oct. 30, 2017, which is herebyincorporated by reference herein in their entirety.

INDUSTRIAL APPLICABILITY

The present disclosure is suitable for a system that detects a shelflabel in a display shelf.

REFERENCE SIGNS LIST

-   1 Stock management system-   2 Shelf label detection device-   10 Camera-   20 Computer (stock monitoring device)-   50 Display shelf-   51 Shelf label-   52 Shelf board-   70 Product-   201 Processor-   202 Input device-   203 Output device-   204 Memory-   205 Storage-   206 Communicator-   211 Shelf label detector-   212 Monitoring area setter-   213 Lacking detector-   214 Output section-   215 Shelf label associating section-   216 Detected shelf label corrector-   261, 262 Communication interface (IF)-   400 Shelf allocation information-   MA Monitoring area-   401 Video obtainer-   402 Manual shelf label setter-   403 Automatic shelf label detector-   404 Shelf label position corrector-   405 Shelf label position determiner-   406 Change detector-   407 Modifier

The invention claimed is:
 1. A shelf label detection device, comprising:an obtainer that obtains first camera video and second camera videoincluding an image of a shelf label arranged in a display shelf; acorrector that corrects a first shelf label position set to generate acorrected shelf label position set by using a second shelf labelposition set, the first shelf label position set including a shelf labelposition that is set by a user in advance in the first camera video, thesecond shelf label position set including a shelf label position for theimage of the shelf label detected from the second camera video throughvideo recognition; and a determiner that determines the shelf labelposition in the second camera video by using the corrected shelf labelposition set.
 2. The shelf label detection device according to claim 1,wherein the corrector calculates a movement amount of the first shelflabel position set based on a size of an overlapping amount between aregion of the shelf label position included in the first shelf labelposition set and a region of the shelf label position included in thesecond shelf label position set, and generates the corrected shelf labelposition set by using the movement amount.
 3. The shelf label detectiondevice according to claim 1, further comprising: a first notificationsection that sends a predetermined first notification when the firstshelf label position set and the second shelf label position set aremisaligned over a predetermined period or longer.
 4. The shelf labeldetection device according to claim 3, wherein the first notificationsection does not send the first notification when a target of themisalignment is an exceptional shelf label that is set in advance. 5.The shelf label detection device according to claim 1, furthercomprising: a second notification section that sends a predeterminedsecond notification when shelf allocation information is misaligned withthe corrected shelf label position set, the shelf allocation informationbeing information on an arrangement position of the shelf label in thedisplay shelf.
 6. The shelf label detection device according to claim 1,further comprising: a second notification section that sends apredetermined second notification when shelf allocation information ismisaligned with the first shelf label position set, the shelf allocationinformation being information on an arrangement position of the shelflabel in the display shelf.
 7. The shelf label detection deviceaccording to claim 1, further comprising: a modifier that modifies thefirst shelf label position set by using the second shelf label positionset when a shelf label position that is present in any one of thecorrected shelf label position set and the second shelf label positionset but is not present in the other is detected over a predeterminedperiod or longer.
 8. The shelf label detection device according to claim7, wherein when a shelf label position that is present in the correctedshelf label position set but is not present in the second shelf labelposition set is detected over the predetermined period or longer, themodifier deletes the detected shelf label position from the first shelflabel position set.
 9. The shelf label detection device according toclaim 7, wherein when a shelf label position that is not present in thecorrected shelf label position set but is present in the second shelflabel position set is detected over the predetermined period or longer,the modifier adds the detected shelf label position to the first shelflabel position set.
 10. A shelf label detection method, comprising:obtaining first camera video and second camera video including an imageof a shelf label arranged in a display shelf; correcting a first shelflabel position set to generate a corrected shelf label position set byusing a second shelf label position set, the first shelf label positionset including a shelf label position that is set by a user in advance inthe first camera video, the second shelf label position set including ashelf label position for the image of the shelf label detected from thesecond camera video through video recognition; and determining the shelflabel position in the second camera video by using the corrected shelflabel position set.
 11. A shelf label detection program causing acomputer to execute a process comprising: obtaining first camera videoand second camera video including an image of a shelf label arranged ina display shelf; correcting a first shelf label position set to generatea corrected shelf label position set by using a second shelf labelposition set, the first shelf label position set including a shelf labelposition that is set by a user in advance in the first camera video, thesecond shelf label position set including a shelf label position for theimage of the shelf label detected from the second camera video throughvideo recognition; and determining the shelf label position in thesecond camera video by using the corrected shelf label position set.